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Abstract — We consider approximations of signals by the ele- 
ments of a frame in a complex vector space of dimension TV and 
formulate both the noiseless and the noisy sparse representation 
problems. The noiseless representation problem is to find sparse 
representations of a signal r given that such representations 
exist. In this case, we explicitly construct a frame, referred 
to as the Vandermonde frame, for which the noiseless sparse 
representation problem can be solved uniquely using 0(N 2 ) 
operations, as long as the number of non-zero coefficients in 
the sparse representation of r is e/V for some < e < 0.5, thus 
improving on a result of Candes and Tao [3]. We also show that 
e < 0.5 cannot be relaxed without violating uniqueness. 

The noisy sparse representation problem is to find sparse 
representations of a signal r satisfying a distortion criterion. In 
this case, we establish a lower bound on the trade-off between 
the sparsity of the representation, the underlying distortion and 
the redundancy of any given frame. 

Index Terms — Frames, sparse representations, redundancy, 
sparsity, distortion 

I. Introduction 

LET r be a complex N dimensional signal and B be a 
basis for <C N . Then it is well-known that r has a unique 
expansion in terms of the elements of this basis. In particular, 
if B is the Fourier basis, then fast algorithms for computing 
the expansion coefficients of r are very well-known. 

Consider now a set T of M > N non-zero signals in an 
TV-dimensional complex vector space W such that T spans 
W. We refer to J as a frame or a dictionary for W. For 
r e W there are possibly infinite ways to represent r as a 
linear combination of the elements of T. In this paper, we 
are interested in sparse representations of r with the lowest 
number of non-zero coefficients (referred to as the Cq norm of 
the representation vector). In fact, sparse representations have 
recently received wide attention because of their numerous 
potential applications. Such applications include Magnetic 
Resonance Imaging, where only a partial set of measurements 
are available to describe an object [1]; compression using 
overcomplete dictionaries; separation of images into disjoint 
signal types, etc. (Please see [5] and the references therein). 

If the signal to be represented is known to have a sparse 
representation, the question of interest is to find the exact 
sparsest representation in terms of the dictionary elements. 
This problem will be referred to as the noiseless coding 
problem. The difficulty of this problem has caused many 
researchers to look for approximations to the solution. Two 
most commonly used methods are the orthogonal matching 
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pursuit (OMP) and basis pursuit (BP). OMP is a greedy 
algorithm, which generalizes the classical orthonormal basis 
algorithm. The OMP algorithm starts with the residual signal 
at step zero set to be the original signal r. Then at each step i, 
the dictionary element that has the highest correlation with the 
residual signal is selected. The residual signal at time i > 1 
is then updated to be the projection of the residual signal at 
time i — 1 on the orthogonal complement space of the subspace 
spanned by the dictionary elements chosen up to and including 
the stage i [14]. In contrast, the basis pursuit (BP) algorithm 
is based on a linear programming approach to the sparse rep- 
resentation problem, where instead of minimizing the number 
of nonzero coefficients in the approximation, minimization of 
the sum of the absolute values of the coefficients (i. e. the 
L\ norm of the representation vector) is the objective [4]. 
Both algorithms can be applied to arbitrary dictionaries, and 
little attention has been paid to the construction of dictionaries 
that support simple algorithms for the computation of sparse 
representations. However, the construction of such dictionaries 
and their importance may be evident from various applications 
[5]. This motivates our studies in this paper, where we consider 
two important cases of this problem namely the noiseless and 
noisy sparse representation problems. 

The noiseless sparse representation problem considers the 
case when r 6 W and it is known in advance that the signal 
r has a sparse representation in terms of the elements of the 
frame T. In this case, the goal is to find a solution to the 
sparse representation problem in real time. This is a problem 
commonly encountered in signal theory. For instance, when 
the underlying frame is the Fourier basis, then the classical 
problem of finding the Fourier expansion coefficients is of 
immense interest. In this case, a question of fundamental 
importance is the fundamental limits on sparsity of r for which 
a unique noiseless representation exists, and the construction 
of the frames that achieve these fundamental bounds and 
support real time solutions. We will provide a solution to this 
problem in this paper. 

The noisy sparse representation problem considers the case 
when r S W is not known to have an exact sparse rep- 
resentation. In this case, the signal r cannot necessarily be 
represented in terms of the elements of the frame T in a sparse 
manner, and any such sparse representation suffers from some 
distortion. The objective in this case is to trade-off sparsity for 
distortion. Let the redundancy of T be i — 1, where r = Mj 
N. We will study the trade-off between sparsity, distortion and 
redundancy, a problem of fundamental importance. Another 
important problem is to construct frames T for which not 
only these trade-offs can be achieved, but also the underlying 
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sparse representations can be found in real time, and this is 
currently being investigated. 

The outline of this paper is given next. In Section [II] 
we provide a solution to the noiseless sparse representation 
problem. We will construct an explicit frame, which we refer 
to as the Vandermonde frame, for which the noiseless sparse 
representation problem can be found uniquely using 0(N 2 ) 
operations, as long as the number of non-zero coefficients 
of the sparsest representation of r over the frame is eN for 
< e < 0.5. We will also argue that e < 0.5 cannot be relaxed, 
without violating uniqueness. In Section [Ell] we consider the 
noisy sparse representation problem and propose a statistical 
approach to this problem. We compute a lower bound on the 
trade-off between sparsity, distortion and redundancy for any 
frame T. Finally in Section [IV] we will make our conclusions 
and provide directions for future research. 

II. The Noiseless Sparse Representation Problem 
As in the previous section, consider a frame T = 
{4>i, 2 , ' ' ■ j0m} °f M > N non-zero vectors that span 
an N dimensional subspace W < C M . Any vector r in 
W can be written in (possibly a non-unique) way as the 
sum of the elements of T . Let r = YlfLi c i4>i be such a 
representation. We define ||r[|o ^ to be the smallest number 
of non-zero coefficients of any such expansion. Also, for an 
arbitrary vector c = (ci, 02, • • • , cm) € C M , we define ||c|| 
to be the number of non-zero elements of c. Thus ||r||o,jF is 
simply the min(||c||o) over all possible expansions c of r as 
above. A main problem of interest is 

• The Most Compact Representation (MCR) Problem: 
Given T a frame spanning W, and r G W find an ex- 
pansion r = YhLi c i4>i f or which c = (ci, c 2 , • • ■ , c M ) 
has minimum ||c||o. 
Let F be the matrix whose rows are the elements of the frame 
T . Then the MCR Problem can be restated as: 



mm 

cSC M 



|c|| s.t. r = cF 



(1) 



This optimization problem is in general difficult to solve. In 
this light, much attention has been paid to solutions minimiz- 
ing ||c||i = Yli=i \ c i\ instead, and then establishing criteria 
under which the minimizing c also solves the MCR Problem 
[1], [9]. 

In this section, we will take a different approach. We will 
construct an explicit frame T for which the following problem: 

• Decoding Problem: Whenever r has a representation c 
with ||c|| = eN, for < e < 0.5, then find c, 
can be solved with a unique answer in running time 0{N 2 ). 

A. Connection with Error Correcting Codes 

In solving the MCR problem using the above approach, 
we make a simple albeit fundamental connection between 
solutions to the MCR Problem and error correcting cod- 
ing/decoding. Such connections have also been made by 
a number of other authors who have realized connections 
between frames and linear codes defined over the field of 
complex numbers [9]. Inspired by this connection and the 



theory of algebraic coding/decoding, we construct frames that 
generalize Reed-Solomon codes using Vandermonde matrices. 
Under the assumption that ||r||o,.F < N/2, a generalized Reed- 
Solomon decoding algorithm (which corrects up to half of the 
minimum distance bound) can find the solution to the decoding 
problem and the MCR problem. Such decoding algorithms 
and their improvements are well-known in the coding theory 
literature. 

Consider a frame T = {0i, 02 > " ' 1 0m } °f M non-zero 
vectors that span an N dimensional subspace W < C M as 
above. Consider: 



V = {d = {d 1 ,d 2 ,-- ■ >d M ) e 



M 

£ 

1=1 



di<t>i = 0} 



The vector space V is clearly an M — N dimensional subspace 
of C M . If r G W can be represented by c with respect to 
the above frame T, then all possible representations of r are 
given by c — V = {c — d | d e V}. Thus the problem of 
finding the sparsest representation of r is equivalent to finding 
d G V which minimizes ||c — d||o- If one thinks of V as a 
linear code defined over the field of complex numbers, and 
of r as the received word, the MCR Problem is equivalent to 
finding the error vector e = c d of minimum (Hamming 
weight) ||e||o over all the codewords d G V. Problems of this 
nature have been widely studied in the language of coding 
theory, however these codes are typically defined over finite 
fields. The main contribution of this paper is the observation 
that complex analogues of the Reed-Solomon codes can be 
constructed from Vandermonde matrices and the associated 
sparse representations can be computed using many well- 
known Reed-Solomon type decoding algorithms. 

B. Vandermonde Frames 

Consider the matrix given below: 
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Z\ 


7 2 
z \ 
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Z2 


7 2 
/ "2 


A = 
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Z3 


-.2 

z 3 




V 1 


ZM 


Z M 



JV-1 



y N-l 
-M 



1 \ 



(2) 



where z%, i — 1,2, ■■• ,M are distinct, non-zero complex 
numbers. The following 

• Condition I: Any arbitrary set of N distinct rows of A 
are linearly independent 
holds. This is clear since any such N rows form a Vander- 
monde matrix with non-zero determinant. 

We define our frame T to consist of the rows of A, i.e. 



1 )forj = l,...,M} (3) 



T = {0j = (1, zj,z?, . . . , zf~ 

and refer to it as a Vandermonde frame. 

Let W be the N dimensional subspace spanned by the 
elements of T . The subspace V, as defined above, is given 
by the vectors d = (d%, ■ ■ ■ , c?m) for which 



d 1 



0, 



(4) 
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where 



M 



i=i 



(5) 



Clearly the subspace V is M — N dimensional. Furthermore 
if w e W and w = J2iLi c i<t>i, th en 



w = (to 1 , w 2 , • • • ,w JV ), 



iV \ 



where 



to 



M 
.7=1 



(6) 



(7) 



The subspace V has the following interesting property. 

Lemma 2.1: For any non-zero vector v S V, we have 
||v||o > AT. Moreover, there exist vectors v G V with 
||v|| = iV + l. 

Proof: Suppose that v S V is non-zero and ||v||o < 
N. Let the nonzero elements of v occur in locations 
{ji> J2, • • • , jw} where w = ||v|| < N. Then rows 
{ Ji j J2 , • • • of matrix A are dependent. This violates 

Condition I. To observe that there are vectors v 6 V with 
|| v l|o = N + 1, we have to only exhibit a linear dependence 
between N + 1 rows of A. This is trivial since any arbitrary 
N + 1 rows of A are linearly dependent. □ 

In the language of algebraic coding theory, the subspace V 
is a maximum distance separable (MDS) linear code of length 
M, dimension M — N and minimum distance N + 1. In fact 
as we will see from our decoding algorithm, this subspace 
provides complex analogues of the Reed-Solomon codes. 

C. Uniqueness of The Sparsest Representation 

Next, let a vector r S W be given and we are given that 
|| r ||o,.F < N/2. We will first prove the following important 
Lemma. 

Lemma 2.2: Given that ||r||o,^ < N/2, the solution to the 
Decoding Problem is unique. 

Proof: Let r = Y,f=i c i<t>j and r = 2~2f=i d j4>j 
be two solutions to the Decoding Problem with ||c = 
(ci,..., CAf )||o < N/2 and ||d = (d u --- ,d M )\\o < N/2. 
Then c - d 6 V and ||c - d|| < ||cj| + j|d|| < N. By 
Lemma 12.11 c d = and c = d. □ 

The importance of Lemma 12.21 follows from the following 
obvious albeit fundamental observation. If in the decoding 
algorithm, we were interested in finding all the representations 
c of r with ||c||o < (N + l)/2, then the solution was not 
necesarily unique. This non-uniqueness can be seen from 
Lemma |2~T1 Because there exists v e V with ||v||o = N + 1, 
one can easily construct two distinct representations of the 
same vector in W both having (N+l)/2 non-zero coefficients 
(provided that (N + l)/2 is an integer). Thus the bound in 
the Decoding Problem cannot be improved assuming that the 
algorithm is to output a unique solution. 

We note that the uniqueness of the solution to the Decoding 
Problem is not necessary when considered from the point 
of view of frame theory. In fact, our proposed decoding 
framework may be readily generalized using well-known 



methods in coding theory to design algorithms that list all 
possible compact representations of a given vector r S W. 
Such algorithms are known as list decoding algorithms in the 
literature [11]. 



D. The Decoding Algorithm 

We next provide a polynomial time algorithm that outputs 
the sparsest representation of r under the assumption that 
H r l|o,.F < N/2. Let r = 2~2jLi c j4>j be an arbitrary rep- 
resentation of r in this frame. A candidate c can be easily 
computed using 0(N 2 ) operations. For example if we let 
cn+i = • • • = cm = 0, then c\, ca, • • • , cn can be computed 
by multiplying the inverse of a Vandermonde matrix (that can 
be once computed off-line) by r, requiring at most 2N 2 opera- 
tions. We fix the representation (ci, C2, • • • , Cm) of r and seek 
to compute the most compact description (ei, e%, - ■ ■ , e\i) of 
r = Y^jLi e -j ( t ) 3 m tn is frame with e = ||(ei, e 2 , • • • ,ejvf)||o < 
N/2. Clearly 



(ci,c 2 ,-" ,cm) =e + d, (8) 
where d = (di, d 2 , ■ ■ ■ , du) £ V. For any i = 1, 2, • • • , let 



M 



and 



M 

M 
3=1 



C 3 Z 3 



then by Equation @), we have 



(9) 
(10) 

(ID 



for % = 1,2, Thus e% i = 1,2, •••,7V can be 

computed using at most 2MN operations. 

Let the nonzero elements of e = (ei,e2,--- , ejw) be in 
iij*2) • " " ! "iw where w < N/2. For j = 1, 2, • • • , w, let X,- = 
Zi and ij = e,. . The following Lemma gives the analogue 
of the Key Equation in Reed-Solomon decoding [7]. 

Lemma 2.3: Define 



a[z] = H(l-X i z), 



i=l 



oo 



then 



w[z] = Sfzjcrfz], 
anywhere in the disk |z| < mini<j<jvf(|^j| -1 )- 



(12) 
(13) 
(14) 

(15) 
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Proof: Although the proof is given in [7], for complete- 
ness we repeat the proof here. Clearly 



a[z] 



Yi 



1—1 



(16) 



Under the assumption of \z\ < mini<j<M(|^j| *)» we have 

1 oo 



1-X t z 4-i 

3=0 



Replacing this in Equation ( TToT l. we have 



uj[z± 
a[z] 



w oo 



rj-l 



(17) 



□ 



Clearly e> = Y i x l~ ■ Thus the result follows. 

Since deg(w[z]) < eA - 1 < A/2 - 1 and deg(<r[z]) = 
eN < A/2 only e 1 , e 2 , • • • , e are needed to compute u>[z] 
and a[z] from the above (for instance by solving a linear 
system of equations for the coefficients of u>[z\ and a[z]). It 
is well-known that this task can be achieved more efficiently 
using the Euclid division algorithm [7]. In fact, letting Si [z] = 
X^=i e? z^ 1 one can write: 

u[z] = Si[z]a[z] mod 

for all z S C. The computation of u>[z] and a[z] can be 
performed using the Euclid division algorithm as described 
for instance in [7] (Section 9, Chapter 12). The number of 
operations required for the execution of this algorithm is 
clearly O(MN). 

Once a[z] and u>[z] are found, we first compute a[z] for 
z i j z 2 > ' ' ' >%/■ This step only requires 0(eN 2 ) compu- 
tations (since the required powers of Zj, j = 1,2, •■• ,M 
must only be once computed off-line). In this way, the roots 
z" 1 ,--- , 1 of a[z] (and hence the locations of non-zero 



elements of e) can be found. The values , 
be found using the formula (attributed to Forney) 



can then 



(18) 



where (j'[z\ is the derivative of cr[z]. 

In conclusion the vector e giving the most compact repre- 
sentation of r can be computed with complexity 0(N 2 ). 

We note that the explicit construction in Sectionllllimproves 
constructively on the required sparsity factor of other existing 
techniques [3] that provide a solution to the MCR problem. 
However, the basis pursuit and OMP algorithms also work for 
the noisy sparse representation problem. It is not immediately 
clear how to solve the noisy sparse representation problem 
for the Vandermonde frames and this topic is currently being 
investigated. 

III. The Noisy Sparse Representation Problem 

The noisy sparse representation problem considers the case 
when r S W is not known to have an exact sparse repre- 
sentation. Then r cannot necessarily be represented in terms 
of the elements of the frame T in a sparse manner, and any 



such sparse representation suffers from some distortion. In 
addressing this distortion, since any scaling of r by a factor 
of a 7^ changes the distortion in a sparse representation of 
r in terms of a set of given vectors by a factor of \a\ 2 and 
does not affect the sparsity, it can be assumed without loss of 
generality, that ||r||2 = \/~N. 

Two classes of noisy representation problems have been 
considered in the literature, namely bounded distance sparse 
decoding (BDSD) and sparse minimum distance decoding 
(SMDD) [12]. These are formulated as: 



mm c 

ceC M 



s.t. ||r-cF|| 2 <<5||r|| 2 (BDSD) (19) 



or 



min I |r — cFll 2 , 



s.t. ||c|| < eN 



(SMDD) (20) 



Both BDSD and SMDD problems have been studied in an C\ 
setting [2], [5] or by using OMP methods [13]. Almost all the 
existing research are based on a worst case criterion, where 
the maximum distortion over all vectors r is the underlying 
measure of performance. 

The SMDD and BDSD problems are intimately related. In 
particular, if the SMDD problem can be solved in polynomial 
time, then so can the BDSD problem. In fact the threshold 
on the value of ||c||o can be decreased from N to in iV 
applications (or in log 2 (N) applications, with a binary search) 
of the polynomial time algorithm for the SMDD problem, until 
the minimum distortion exceeds the required threshold for the 
BDSD problem. 

For a given r, the SMDD problem finds the closest (in 
the Euclidean distance sense) sparse representation Yliei Ci< ^i 
(with \T\ < eN). In order to quantify the performance of 
SMDD, we propose an average distortion approach. Such 
measures are motivated by information theory and by the 
fact that we seek frames for which the SMDD works well 
for typical signals. In fact, if a given vector r is selected 
according to a given distribution on the hypersphere of radius 
VW, an appropriate frame must be designed to reduce the 
average distortion of SMDD. If no knowledge of r is at hand, 
it is natural to assume that r is distributed uniformly on the 
complex hypersphere of radius y/W centered at the origin, 
and this assumption will be used throughout the rest of this 
paper. In formal words, for any frame T, and sparsity factor 
< e < 1, our measure of performance of T is given by 

D(T) = ^E r min||r-cF|| 2 , 

where the minimum is taken over all representations c of 
r with ||c||o < eN and the expectation is for r uniformly 
distributed on the dimensional complex hypersphere of 
radius y/N centered at the origin. 

A. Trade-off Between Sparsity, Redundancy and Distortion 

Let M — rN, where r — 1 is the redundancy, and let L = 
eN, where e is the required sparsity. Let T = (^). 

Consider all the L-dimensional subspaces of W that are 
spanned by all subsets of size L of {4>j}jei k - There are < 
T distinct L-dimensional such subspaces denoted by {Pk, k = 
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1, 2, • • • , T*}. Given a vector r on the N dimensional complex 
hypersphere, the SMDD algorithm find the closest Vk,k = 
1, 2, • • • , T* to r. In other words, it minimizes | |r — Up fc r| | 2 , 
where Hp fc is the projection operator onto Vk- 

Using this geometric interpretation, we will find a lower 
bound on the distortion as a function of sparsity and re- 
dundancy of any frame T . To this end, we define an L- 
dimensional complex generalized cap of radius yfp around an 
L-dimensional plane V k as 



GC L (p,T k ) = {x£ § : | jx — rip fc x|| < p} 



(21) 



where S is the N dimensional complex unit hypersphere 



{xG 



^N . 



!}■ 



If we are only interested in the radius of the generalized cap, 
but not the specific plane, we will use the notation GCl(p)- 
In order to calculate the quantity of interest, 
iE r ( mim<fe<T, ||r - Hp fc r|| 2 ) for uniformly 
distributed on § , we need to know the distribution of 
min fe d 2 (x,'Pfe), where d 2 (x,"Pk) = ||x- II-p fc x|| 2 and x is 
uniformly distributed on § . Clearly, for any given x € S^, 
we have 

P(mind 2 (x,7> fc ) < rf) 

= P(There exists a plane within distance ^Jrj of x) 
= P(x is in the area covered by the generalized 
caps of radius y/rj) 



xe |J GC L (V k , v ) 

k=l 

Since < T 

T, 

xe |J GC L (V k ,ri) 



k=l 



^E 1 

fe=i 



xeGC L (V k ,v) < T 



^A(GC L ( V )) 



Thus 



f(mmd 2 (x,V k ) < rj) <T 

k 



and 



'(min<r(x,7> fc ) > rf) > max 1 

k 

In order to bound 



A{S N ) 

A (GC L (y)) 
A(S N ) ' 

A(GC L ( V )) 



(22) 



T 



A($ N ) 



(23) 



E(mmd 2 (x,V k )) = [ P{mmd 2 (x,V k ) > rfidr) 
k J k 

We will establish a lower bound on the right hand side of 
Inequality (l23l . by estimating the value of 77 for which = 
l-TA(GC L (r,))/A(S N ). 

Clearly H-p satisfies: IT£, = H-p and lip = H-p. It is well- 
known that the projection matrix H-p can be diagonalized as 
Up = U DU*, where U is a unitary matrix and D is a diagonal 
matrix with l's as the first 1, 2, • • • , L and O's as the rest of 
N — L diagonal entries. Also we note that ||x — ]Tpx|| 2 = 




Fig. 1. A Generalized Cap in Mr with N = 3, L = 2 

||i7*x — DU*x\\ 2 , and C/*x is just a unitary transformation of 
x that has the same uniform distribution as x. 

Let w be a vector uniformly distributed on the unit hy- 
persphere. One way to generate w is to take a complex zero- 
mean Gaussian vector z ~ jV c (0, In), where In is the identity 
matrix, and let w = [8] (Thm 1.5.6). Clearly w has the 
same distribution as U*w, and thus ||w — il-pw|| 2 has the 
same distribution as ||w — Dw|| 2 = |wl+i| 2 + ■ 

It is easy to see that 

|| W _ Dwll 2 = |z , L+ , 1 J 2 + '" + |ZjV J 2 



\w N \ 



+ ■ ■ ■ + \z N \ z 

where each ~ A/" c (0, 1) is a complex Gaussian random 
variable. It is well known [8] (Thm 1.5.7) that this fraction 
has (3(N — L, L) type distribution. Thus 

- N - L -\l-x) L - l dx 



|w - Dw|| 2 <p)= [ 
Jo 



r t 1 



where C 



rjjV) 



N,N-L T(N-L)T(L) 



But also 



|w - Dwll 2 < p) = 



1 ^ = A{GC L ( P )) 



'GC L (p) 

Therefore for an L-dimensional generalized cap in C N we 
have 



A(GC L (p)) _ [P T(N) 



N-L-l 



(l-x^dx 



A(S N ) Jo T(N - L)T(L) ' 



(24) 



We now prove a number of technical lemmas. 
Lemma 3.1: 

rp 

x' 

r p / \ N-L-l 



f 

JO 



- If - L - 1 (l-x) L - l dx< 



X 



V 1 ~\~ T 1 X 



N-L-l' 



dx 



Proof: 

(l-x) L - 1 [l + 



L - 1 
N-L- Y 



N-L-l 



< 



e -(L-l)x e (L-l)x = 1 



Hence we have 

{l-x) L - x < 



1 



1 



L-l 



N-L-l 



N-L-r") 

Substituting this to the integral gives the desired result. □ 
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Lemma 3.2: Let f(x) = — , then f(x) attains its 

1+ N -L-l X 

maximum for x € [0, p] at p. 

Proof: By direct computation, we have 

1 



/'(*) = 



f1 j L -! a-12 



> 



for x € [0, p]. Thus /(x) is an increasing continuous function 
that attains it maximum at p. □ 
We conclude from the above lemma that 

N-L-l / \ N-L-l 

dx < i -. 

1 + 



Lemma 3.3: 

T(N) 



< (N-l)2 (N - 2)H (-^) 

(N-iy. 

T(N — L)T(L) (N - L - 1)\(L - 1)\ 

< 



T(N-L)T(L) 
Proof: 

T(N) 



L-l, 
(N - l^-^te) 



□ 

By combining these results, we obtain the following bound: 

A(GC L (p)) <T(N _ 1) { n-2)h(^) 

-i , Li - ) = A (^) P5) 

1 + N-L-\P' 



We define 

Kc (7V) 4 T^^t (JV- l)iv^T2 _ ^ e T^T i:, V — 7, (26) 



then 



Lemma 3.4: For any frame of dimension N and size M 
over C, and for a sparse representation over for x uniformly 
distributed on the unit hypersphere § w , with at most L = eN 
nonzero coefficients, we have 

. For L = 0, P(min fe cP(x, V k ) > p) = 1 for any < p < 
1. 

. For 1 < L < N - 2, the equality A(p, N) = 1 is attained 
at 



P = Po(iV) = 



«c(JV) 



1 



L-l 
N-L- 



jK c (N) 



(27) 



where 

K c (N) = TT^=i (iV-l)iv^T 2~^T 



-1 _ N-2 m L-l j 

1 H (n-2> 



and < p (A r ) < 1. 

For L = JV - 1, TA(GC L (p)) /^(S^) = 1 at 

Po(iV) = 1 - 1 1 - f 

For L = N, ¥{min k d 2 (x,V k ) > p) = for any < 
P<1- 



Proof: The results for L = and L — N are obvious. 
Thus without loss of generality, we can assume 1 < L < 
N-l. Hence T > 1. For 1 < L < N - 2, we first claim that 



L - 1 



N-L-l 



k c (N) < 1 



(28) 



Clearly, 



N— 2 

2-«=r= 



T ff (#-r|) 



i- 1 

iV-2 



N-L-l 
N-2 



thus 



L - 1 



N-L-l 



Kc (N) = T^T^T (7V_l)lv^rT 



L-1\ N - L - 1 {L-l 



N-2 



N-2 



The claim is proved since T N - L - 1 < 1, (TV-l)"-'-- 1 < 1, 

N-2 

and (^Ei) JY_i_1 < 1. The value of p (iV) and the fact that 
Po{N) > now follows from Equation ( T25l l. Since 

TA{GC L (p a (N)))/A(S N ) < A(p (N),N) = 1, 

we have ^(GC L (po(A r )))M(S Ar ) < 1 and thus p (iV) < 1 
as claimed. 

For L = N — 1, we directly calculate 

A(GC L ( P )) _ f rAT 1W1 _^_ 2j __, ,„ ^jy-x 



^) yo (^-i)(i-»r-^=i-d-p) 

Therefore TA(GC L (p))/A(S N ) becomes 1 at 

1 \ ^ r 



po(7V) = 1-1 



T 



and < po(iV) < 1. 

We now prove the main result of this section. 



□ 



Theorem 3.5: For any frame T over C of dimension N, 
redundancy r — 1 = M/N — 1, for sparsity factor e = Lj 
N, and for r uniformly distributed on the TV-dimensional 
hypersphere of radius y/N, we have 

• For L = 0, we have D(J-) = 1, 

. For 1 < L < N - 2, we have 

D(T) = — E r ( min I |r - Hp t r| I 2 



where, 



and 



> Po(N) 
Po(N) 



Po(N) 2 1 
k c (N) N-L 

Kc{N) 



1 - W^-MN) 



-1 , -1 Af-2 tt f L—\ \ 

K C (N) = T N-L-l (JV - 1) N-L-l 2—N=L=1 H (n=2) . 

For L = N — 1, 

> Po(iV) - tL(JV) - 1 + 1(1 - PoW)^ 
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where 



Po(N) = 1 



T 



For L = N, we have D(T) = 0. 



Proof: The results for L = and L = A are obvious. 
Let x = r/y/N, then 

D(JT) =E(mind 2 (x,-p fc )) = / P(mind 2 (x, "P fc ) > v)dv 
k J k 



and we have previously proved that 

P(nW 2 (x,7\.) > r,) > max (l - T ^ N ( 

By applying Inequality ( 1251 ). we have 

P(mind 2 (x,'Pfc) > rf) > max(l - A(?7, N),0) 



Combining the above and by applying Lemma 13.41 . we have 

rPo(N) 



rPo(") 

D{T)> / (l-A(r),N))dr] 
Jo 



A(GC L (n)) 



for 1 < L < AT - 2 and 
for L = N - 1. Thus for 1< L < N - 2 



D{T)> l-T(N -1)2 (N - 2)H \^) 

Jo 

, Af-L-l" 
/•Po(JV) 

= Po (N)- 



T (A - l)2 (Ar - 2)H (^) 
?7 



1 



N-L-l 



di) 



(29) 



We will next bound the last integral. Let 

1 1 W-2 u ( L-l \ n 

y = J 1 N-L-l /]\? — l)N-t-l2iV--L-l- H VN^2j ' 

1 + i^zV? 



(30) 



where k c (N) is defined as in Equation ((26). Clearly y ranges 
from to 1 in this case. Also 



Kc(N)y 



(31) 



Since by Equation (J28j, « c (A r ) Jv ^i 1 < 1, and < y < 1, 
the denominator of the above is not equal to within the region 
of integration. Now we note that 



K C (N) 



r^y = g(y)dy. (32) 



Also 



< 



K C (J\0 Po(iV) 2 



(l - Kc(A^) 



L-l 
N-L-l 



«c(W) 



(33) 



since g'(y) > for y € [0,1]. Thus the last integral in 
Equation d29l is bounded above by 



1 Po(AQ 2 
/o «cW 

which gives us 



> N - L - X dy 



Po(iV) 2 



l 



D{T) > po(N) 



k c (N) N-L 

Po(AQ 2 1 
k c (N) N-L 



(34) 



(35) 



and the result for 1 < L < A - 2. 
For L = N — I, 



D{T) > max fl-T 



A(GC L (v)) 



)di 1 



A(S N ) 

Jo 

and the result follows easily by direct integration. □ 

We note that all the bounds attain values in [0, 1], since we 
have integrated a function that takes values in [0, 1] over a 
region [0,po(N)] C [0,1]. Theorem 13.51 gives a fundamental 
limit on average distortion that any frame over C has to satisfy. 
We now fix r and e and let N — » oo. The following asymptotic 
result follows: 

Corollary 3.6: For any frame T over C of dimension N, 
redundancy r— 1 = M/N—l, for sparsity factor e = L /N and 
for r uniformly distributed on the A^-dimensional hypersphere 
of radius \/~N, as N — > oo, we have 



«o(l - e) 
1 - e«o 



(36) 



where 



k = 2"— H [r) e i±- C (37) 
Proof: The result follows by replacing A/ = rN and 
L = e N in the statement of Theorem 13.51 and using 



1 



M+l 



< T < 2 



rNH[ 



and 



lim k c (N) = kq(1 — e). 

AT— »oo 



□ 

It is noteworthy that the above asymptotic corollary can be 
proven by combining a result of Sakrison [10] with the proof 
method of [6], although the setting and the topic of our paper 
is very different from these papers. 
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IV. Conclusion 

In this paper, we considered approximations of signals 
by the elements of a frame in a complex vector space of 
dimension N. We formulated both the noiseless and the noisy 
sparse representation problems. For the noiseless representa- 
tion problem, it is known in advance that the signal r has a 
sparse representation in terms of the elements of the frame 
T . In this case, the goal is to find a solution to the sparse 
representation problem in real time. We provided a solution 
to this problem, by explicitly constructing a frame, which 
we referred to as the Vandermonde frame, for which the 
noiseless sparse representation problem can be solved uniquely 
using 0(N 2 ) operations, as long as the number of non-zero 
coefficients in the sparse representation of r is eN for some 
< e < 0.5. This result improves on a result by Candes 
and Tao [3]. We also showed that e < 0.5 cannot be relaxed 
without violating uniqueness. 

For the noisy sparse representation problem, we considered 
the case when the signal r cannot be represented in terms of 
the elements of the frame T in a sparse manner and noted that 
any such representation suffers from distortion. In this case, we 
established a lower bound on the trade-off between sparsity, 
distortion and redundancy. Our future research will focus on 
constructing frames for which not only these trade-offs can be 
achieved, but also the underlying sparse representations can 
be found in real time. 
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